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1.  Introduction 


Fast  Interactive  Integrated  Modeling  and_  Strategy  Design  (FASTIMS)  is  Icosystem’s 
integrated  framework  for  decision  support  for  the  intelligence  community.  In  this 
research,  we  used  FASTIMS  to  advance  the  rapid  assessment  of  technological 
capabilities  of  potential  adversaries.  With  FASTIMS,  intelligence  units,  analysts  and 
decision-makers  are  able  to  create  models,  perform  simulation  validation  and  calibration 
interactively  to  evaluate  evolving  technologies  of  potential  adversaries,  discover  key 
indicators,  and  quantity  uncertainty.  FASTIMS  enables  analysts  to  identify  data  needs 
and  design  and  test  strategies  to  determine  performance  characteristics,  capabilities,  and 
vulnerabilities  of  adversaries  in  a  matter  of  hours  or  days. 

This  report  describes  Icosystem’s  initial  efforts  to  develop  main  components  for  the 
FASTIMS  framework,  a  generalized  agent-based  pathways  model.  Pathways  processes 
are  modeled  as  dynamic  systems  of  interacting  agents.  Here  “agent”  refers  broadly  to 
bundled  data  and  behavioral  methods  representing  an  entity  constituting  part  of  a 
computationally  constructed  world  of  pathways.  Examples  of  possible  agents  include 
individuals  (e.g.  military,  political  and  scientific  personnel),  social  groupings  (e.g. 
government  agencies,  research  organizations/universities,  Finns),  processes  (e.g.  research 
activities,  development  activities,  production/employment  activities),  technology  based 
entities  (e.g.  research  programs,  development  programs,  production  programs),  and 
physical  entities  (e.g.  infrastructure,  equipment,  material  resources).  Thus,  agents  can 
range  from  active  data  gathering  decision-makers  with  sophisticated  learning  capabilities 
to  passive  world  features  with  no  cognitive  functioning.  Moreover,  agents  can  be 
composed  of  other  agents,  thus  pennitting  hierarchical  constructions  in  which  the  lowest 
level  agents  are  characterized  by  continuous  and  discrete  event-variable  dynamics  and  the 
highest  level  agents  by  heuristically  based  decision-making  mechanisms.  Agent-based 
techniques  are  used  to  handle  heterogeneity  in  behaviors  and  domain  descriptions 
associated  with  Pathways  Behaviors.  For  example,  our  agent-based  dynamic  pathways 
model  will  be  capable  of  addressing  challenging  modeling  issues  such  as  modeling 
asymmetric  information,  strategic  interaction,  expectation  formation  on  the  basis  of 
limited  information,  mutual  learning,  social  interactions  etc.  The  advantage  of  the  agent- 
based  representation  is  its  capacity  to  retain  all  information  associated  with  the  variability 
and  interdependency  between  attributes  of  agents  which  might  otherwise  become  lost  if 
aggregate  quantities  were  formed  directly  from  individual  data.  Another  key  advantage  of 
Agent-Based  modeling  techniques  over  more  standard  approaches  (e.g.  System 
Dynamics  techniques)  is  that  events  are  driven  solely  by  agent  interactions  once  initial 
conditions  have  been  specified.  Thus,  rather  than  focusing  on  equilibrium  states  of  a 
system,  the  idea  is  to  watch  and  see  if  some  form  of  equilibrium  develops  over  time.  The 
objective  is  to  acquire  a  better  understanding  of  a  system’s  entire  phase  portrait,  i.e.  all 
possible  equilibriums  together  with  corresponding  basins  of  attraction. 
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2.  Model  Overview 


We  briefly  outline  the  Icosystem  pathways  model,  which  is  now  a  detailed 
conceptual  model  for  which  the  software  architecture  has  been  implemented  to  the  point 
that  a  simplified  end-to-end  model  can  be  run.  In  addition,  the  model  can  import  data 
coming  from  the  Booz  Allen  Hamilton  Agent-Based  social  network  model  of  Leadership. 
The  Icosystem  modeling  effort  is  inspired  in  part  by  the  model  described  in  the  Bodnar 
text  Warning  Intelligence  for  the  Information  Age  distributed  at  the  PAINT  kick-off 
meeting  in  September  [1],  Essentially  the  pathways  model  tracks  the  progress  of  a 
research  program  through  research,  development,  production,  and  finally  employment. 

The  model  contains  six  types  of  agents:  Environment,  Individuals  (e.g.  military, 
political  and  scientific  personnel),  social  groupings  (e.g.  government  agencies,  research 
organizations/universities,  firms),  processes  (e.g.  research  activities,  development 
activities,  production/employment  activities),  technology  based  entities  (e.g.  research 
programs,  development  programs,  production  programs),  and  physical  entities  (e.g. 
infrastructure,  equipment,  material  resources). 

The  environment  contains  all  other  agents;  it  stores  a  socio-political,  military  and 
economic  profile  of  a  geo-political  region  of  interest  that  represents  actual  environment  in 
which  Individuals  within  Social  Groupings  reside  and  perform  Research,  Development 
and  Employment  activities  on  a  variety  of  technology  based  programs.  These  programs 
are  designed  to  achieve  strategic  political  and  economic  goals  of  the  environment’s 
governmental  leadership.  To  execute  these  goals;  scientific,  military  and  political 
leadership  jointly  sets  up  a  series  of  research  and  development  programs  that  when 
executed  will  lead  to  the  country’s  technological  superiority  in  its  geopolitical  sphere  of 
interest. 

The  technology  based  entities  -  programs  are  characterized  by  the  following 
attributes:  funding  resources,  length  of  the  program  and  number  of  research  tasks-goals 
that  the  program  needs  to  address  in  order  to  be  successful.  The  program  agent 
periodically  generates  programs  which  are  allocated  to  different  Social  Groupings  (e.g. 
research  institutions,  universities,  firms  etc.)  based  on  the  level  of  expertise  of  a  particular 
institution  that  are  required  to  successfully  address  the  program  goals  (e.g.  if  the  program 
is  allocated  to  a  research  institution  which  has  level  of  expertise  that  is  less  than  nominal 
then  probability  of  completing  the  program  successfully  within  the  given  time  frame  may 
be  lower  than  for  a  research  institute  with  higher  rating).  At  this  stage  of  modeling  we 
are  assigning  the  programs  to  different  institutions  randomly  (note:  we  plan  to  extend  the 
allocation  methodology  using  market-based  assignment  methodologies  in  the  future). 
After  the  program  has  been  allocated  to  a  social  grouping  or  to  multiple  social  groupings 
we  simulate  the  life  of  the  program  in  the  research  environment.  That  is,  within  the 
research  institution  there  is  a  need  for  personnel  with  a  certain  level  of  skills  necessary  in 
order  to  accomplish  a  required  set  of  research  tasks  before  the  program  can  move  to  its 
development/employment  stage.  The  perfonnance  of  individuals  from  the  social 
grouping  working  on  the  program  affects  the  speed  of  reaching  program  goals  (that  is, 
executing  -  completing  all  tasks).  On  the  other  hand,  the  individual  perfonnance  is 
influenced  by  many  factors  such  as  individual’s  level  of  skills,  available  funding  - 
financial  resources  needed  to  conduct  research,  physical  resources  and  social 
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collaboration.  Our  model  captures  the  dynamics  of  these  factors.  In  addition,  if  the 
required  personnel  are  not  available  upon  arrival  of  the  program  to  a  social  grouping 
agent,  the  research  (or  development  employment)  institution  must  recruit  required 
personnel  before  starting  to  work  on  the  program.  Also,  if  the  personnel  does  not  have 
adequate  level  of  skills  required  to  execute  the  program  tasks  in  the  projected  time  these 
personnel  must  be  trained  so  that  the  program  goals  can  be  met  (e.g.  personnel  on  the 
project  are  Ph.D.  or  M.S  students  -they  will  gain  the  adequate  level  of  skills  while 
completing  their  educational  requirements).  Recruiting  and  Training  processes  can  be 
completed  domestically  or  abroad  resulting  in  different  level  of  obtained  skills.  Our 
model  captures  the  affects  of  recruiting  and  training  on  the  overall  dynamics  of 
program/tasks  execution.  The  metrics  indicating  progress  made  by  a  social  grouping 
during  the  research  phase  are  expressed  in  terms  of  the  rate  of  generation  of  conference 
papers,  journal  papers,  patent  applications  or  requests  for  additional  funding  in  order  to 
expend  research  activities. 

The  results  from  conference  papers,  journal  papers  and  patents  are  used  to  initiate  the 
development  stage  of  the  program  life  which  goes  through  a  similar  cycle  as  in  the 
research  stage.  That  is,  potential  development  programs  are  assigned  to  an  institution 
which  allocates  individuals  with  certain  skills  to  work  on  the  development  program.  The 
level  of  skills  of  individuals  within  a  social  grouping,  physical  resources  and  funding  are 
allocated  to  the  institution  affect  dynamics  of  development  program  execution.  The 
outputs  of  the  development  stage  are  blue  prints  -  designs  that  when  sent  through  the 
employment  stage  may  result  in  a  usable  product. 

The  employment  stage  is  characterized  by  similar  dynamics  as  research  and 
development  stages.  The  employment  stage  is  where  the  ultimate  intention  of  the 
program  becomes  clear  (that  is,  whether  it  was  intended  for  civil  or  nefarious  purposes). 
The  model  will  support  the  estimation  of  the  probable  employment  path  of  a  program  at 
various  stages  of  the  pathway.  It  will  also  permit  exploring  a  variety  of  targetable 
parameters  for  strategy  and  probe  design  (e.g.  disrupting  social  network  between 
collaborators  working  on  a  program,  influencing  recruiting  and  training  of  individuals 
within  social  groupings,  influencing  delivery  of  equipment  required  to  do  research  and 
development,  etc...).  The  model  will  be  capable  of  providing  feedback  to  the  other 
PAINT  teams  via  a  variety  of  metrics  as  well  as  interacting  with  Leadership  models 
developed  by  other  PAINT  teams.  A  specific  example  of  such  feedback  would  be 
metrics  associated  with  different  types  of  resource  requirements  and  utilization  (e.g. 
requirements  and  use  of:  funding  resources,  equipment,  personnel  resources  and  they 
collaborations  during  research,  development  and  employment  stage. 

3.  THE  GENERIC  MODEL  AGENTS 

Generalizing  from  the  process  description  above,  each  of  the  components 
specifying  the  Pathways  processes  will  be  thought  as  a  single  agent  dynamically 
interacting  with  the  other  ones  following  prescribed  relationships.  The  system  agents  are 
represented  as  hybrid  systems  that  involve  both  continuous  valued  and  discrete  variables. 
In  general,  analyzing  such  complex  dynamics  is  difficult  from  an  analytical  point  of  view 
as  solutions  might  not  exist  in  closed  form.  Because  of  these  features,  we  investigate  its 
dynamics  by  means  of  an  agent-based  approach.  That  is,  within  an  individual  agent, 
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behavioral  decisions  may  be  done  by  evaluation  of  their  dynamics  described  by  general 
hybrid  dynamic  models  (GHDS).  However,  the  system  level  behavior  is  then  determined 
by  running  dynamics  describing  the  interactions  among  agents.  The  integration  of  GHDS 
approaches  to  describe  the  dynamics  of  individual  agents  together  with  agent-based 
simulation  gives  us  a  mean  to  study  dynamics  evolution  of  large  scale  interconnected 
systems  in  a  natural  way.  We  now  take  a  closer  look  at  the  structure  of  agents  and  agent- 
based  simulation  of  Pathways  processes. 

3.1. The  Pathways  Processes  as  a  General  Hybrid  Agent-Based  Dynamic  System 
Automaton  Model 


The  key  component  of  the  program  agent  is  a  new  dynamic  model  of  tracking 
the  progress  of  a  research  program  tasks  through  research,  development,  production,  and 
finally  employment.  The  model  illustrates  how  the  perfonnance  of  individuals  with 
certain  level  of  skills  from  the  social  grouping  working  on  the  program  affects  the  speed 
of  reaching  program  goals  (that  is,  executing  -  completing  all  tasks).  On  the  other  hand, 
the  individual  perfonnance  is  influenced  by  many  factors  such  as  individual’s  level  of 
skills,  available  funding  -  financial  resources  needed  to  conduct  research,  physical 
resources  and  social  collaboration.  Our  model  captures  the  dynamics  of  these  factors. 

The  program  tasks  execution  is  fonnulated  as  a  Generalized  Hybrid  Dynamical 
System  -  the  system  fonnalism  for  combined  discrete/continuous  modeling  (Brocket, 
1993).  This  formalism  is  a  combination  of  two  analytical  approaches:  1)  the  discrete 
event  automaton  describing  discrete  phenomena  corresponding  to  discrete  states  and 
dynamics  such  as  the  dynamics  associated  with  a  sailor’s  rating  and  pay  grade  transitions; 
and,  2)  the  differential  equations  system  describing  the  program  tasks  execution  which  is 
formulated  as  a  closed  loop  nonlinear  control  system.  It  is  described  using  a  set  of  first 
order  coupled  differential  equations  where  causal  relationships  between  internal  and 
exogenous  variables  that  influence  the  task  execution  are  modeled  in  a  form  of  simple 
linear  and  nonlinear  feedback  dependencies.  In  simulation,  the  two  parts  alternate  in 
model  execution.  While  the  discrete  part  executes  the  state  transition  at  the  event  times, 
the  continuous  part  computes  the  state  trajectories  in  between.  The  events  define  discrete 
changes  of  the  continuous  input  values.  We  transform  the  event  segments  to  piecewise 
constant  segments  to  accomplishes  this.  For  example,  for  the  program  agent  dynamics  the 
most  frequent  event  transitions  (e.g.  workload  change)  occur  every  seven  days  while  the 
continuous  part  describing  a  task  progress  to  such  an  event  (i.e.  a  piecewise  constant 
segment)  computes  state  trajectories  for  each  day  during  a  week  of  research,  development 
or  employment  phases. 

The  GHDS  framework  provides  a  means  to  specify  the  system  agent,  using 
mathematical  formalism  defined  for  a  hybrid  automaton  model  (Brocket,  1993).  A 
hybrid  automaton  is  a  system  (Brocket,  1993) 

=  (&iir,Z,  n,  £,<!>,  r)  (1) 

Q  is  the  finite  set  of  discrete  states  with  time  based  switching  points  indicating  the  time 
at  which  a  particular  program  transition  occurs  assuming  that  all  necessary  requirements 
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for  such  a  transition  are  met  (e.g.  research  stage  transitions  to  development  stage  )  ),  SJT 
is  the  set  of  continuous  states,  and  n  is  the  finite  set  of  discrete  events.  The  finite  set  of 
edges,  £c()x2®"  xnx{9T  -^-9 V}xQ,  models  the  discrete  event  dynamics  of  the  system. 
An  edge  E  3  e  =  (qe,Xe,Ve,re,q'e)  is  enabled  when  the  discrete  state  is  in  <?e(e.g.  a  1st  task  of 
a  research  program  allocated  to  a  research  institution)  and  continuous  state  is  in  Xe  (e.g. 
current  level  of  task’s  progress).  When  the  transition  is  taken,  the  event  VeeYl  is 
accepted,  the  continuous  state  is  reset  according  to  map  r  .  This  mapping  can  simply 
restart  the  continuous  state  from  a  new  set  of  initial  conditions  (e.g.  the  last  values  of 
such  states  at  the  final  time  while  in  qe  stat).  Then  the  system  enters  discrete  state  qe 
(e.g.  2nd  task  of  the  research  program  allocated  to  the  same  research  institution).  In  this 
representation  of  discrete  event  automaton  we  account  not  only  for  the  order  in  which 
events  occur  but  also  for  event  occurrence  times  which  leads  us  to  a  timed  discrete  event 
system  model.  In  addition,  at  any  given  time  state  q,  the  occurrence  of  an  event  at  a 
given  state  can  lead  to  several  possible  next  states.  In  our  model  we  can  select  one  of 
these  next  states  according  to  some  perspective  set  of  rules.  We  also  employ  a  discrete 
event  controller  to  model  a  situation  in  which  personnel  can  be  abruptly  “redirected”  in 
their  process  of  skill  set  build  up  due  to  new  strategy  initiatives.  A  discrete  event 
controller  for  DES  is  an  external  discrete  event  control  that  disables  some  events  and 
enables  the  others. 

By  modeling  a  program’s  transition  dynamics  as  a  discrete  event  system,  an 
optimal  probing  and  strategy  trajectory  (e.g.  disrupting  social  network  between 
collaborators  working  on  a  program,  influencing  recruiting  and  training  of  individuals 
within  social  groupings,  influencing  delivery  of  equipment  required  to  do  research  and 
development,  etc...)  can  be  derived.  Finally,  £  =  is  the  collection  of  constituent 

dynamical  systems  where  each  =  \xq,Tq,Uq,<bq\  is  a  closed  loop  dynamical  system  with 
inputs  describing  a  task’s  progress  corresponding  to  a  particular  discrete  state  q.  Here, 
theA?  <=9t"  are  the  continuous  state  spaces  and  <t> q  arc  called  the  continuous  dynamics,  Tq 
transition  system.  Continuous-  and  discrete-time  transition  systems  denote  the  cases 
where  Tq  =  $R  (or5R+)  and  rq  =  z  (or  z+)  respectively1.  is  the  set  of  piecewise 

constant  inputs  u  e  U q(t,X  At))  (e.g.  control  input  determines  how  quickly  a  task’s  goal 

can  be  reached  with  available  personnel  assigned  to  work  on  the  task).  The  continuous 
state  evolves  according  to  the  closed  loop  differential  inclusion 

Xq(t)eFq(q(t),Xq(t),Uq(t,Xq(t)))- 

In  simulation,  the  two  parts  alternate  in  model  execution.  While  the  discrete  part 
executes  the  state  transition  at  the  event  times,  the  continuous  part  computes  the  state 
trajectories  in  between.  The  events  define  discrete  changes  of  the  continuous  input 
values.  We  transform  the  event  segments  to  piecewise  constant  segments  to  accomplish 
this. 


1  The  symbols  R,  R+,  Z,  and  Z+  denote  the  reals,  nonnegative  reals,  integers,  and 
nonnegative  integers,  respectively. 
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3.2  Analysis  of  the  Model:  Stability  Analysis 

The  GHDS  framework,  Equation  (1),  postulates  certain  basic  behavioral  actions 
on  the  part  of  the  actors.  What  is  important  in  this  description  is  that  a  particular  task’s 
overall  performance  during  the  research,  development  of  employment  stages  become 
significantly  altered  if  the  strength  of  any  one  of  these  factors  becomes  great  enough 
(exceeds  the  critical  threshold).  In  this  case,  the  system  may  become  unstable  to  the 
“stimulus”  being  imposed,  for  example,  the  critically  undermanned  program  increases  the 
expected  work  per  personnel  assigned  to  the  program  which  eventually  results  in 
performance  degradation.  It  is  therefore,  important  to  know  the  range  of  values  for  which 
the  task  dynamic  model  is  stable,  and  from  the  point  of  view  of  the  decision  maker,  it  is 
important  to  know  the  range  of  values  of  the  parameters  at  which  the  system  is  most 
sensitive  to  change,  so  that  a  timely  policy  initiative  may  be  made  to  effect  a  desired 
change  and  also  to  be  aware  when  an  unwanted  change  may  be  likely  to  occur.  In  other 
words,  for  which  values  of  the  parameters  of  the  system  should  the  decision  maker  be 
alerted  to  a  likely  significant  change  of  system  performance  and  in  what  direction  is  this 
change  going  to  occur?  Also,  what  “perturbation”  must  be  imposed  upon  the  system  to 
affect  its  performance  in  a  desired  way?  The  proposed  mathematical  formalism  for 
modeling  Pathways  processes  allows  us  to  address  the  above  mentioned  issues  by 
applying  mathematical  stability  analysis  for  such  a  hybrid  interconnected  time-varying 
dynamics  system  (Garagic  et  ah,  2006). 


4.  Model  Parameters  and  Data  Requirements 

As  with  any  modeling  approach,  model  calibration  against  real-world  data  is  a  key 
challenge.  In  order  to  make  this  process  more  efficient,  we  have  provided  a  list  of  ideal 
data  requirements.  These  parameters  are  of  different  nature:  some  of  them  are  currently 
implemented  in  the  model  and  their  possible  ranges  were  determined  in  consultation  with 
experts.  Others,  however,  are  kept  variable  due  to  scarce  knowledge  about  their  real- 
world  value  or  usage. 

Inputs  from  a  Leadership  Model: 

•  network  topology  of  scientific,  military  and  political  social  networks 
(nodes/vertices  =  scientists,  politicians,  military  personnel;  edges  =  nodes’ 
domestic  and  foreign  connections) 

•  number  of  connections  among  a  person’s  friends  -  a  measure  of  cliquishness  of  a 
group. 

•  strength  of  connections 

•  associations  of  nodes  to  a  social  network  with 
research/development/production/political/military  institutions  (i.e.  person  A 
works  in  University  of  Teheran) 

•  capabilities  -  level  of  skills  of  individuals  (nodes/vertices) 

•  number  of  publications:  conference  papers,  journal  papers  and  patents  (for  a  given 
node) 
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Information  about  the  Program(s)  personnel  or  teams  (from  scientific/political/military 
social  networks)  is  currently  working  on.  Attributes  of  the  program  would  include: 

•  funding 

•  duration 

•  collaborative  requirements 

•  type  of  research 

•  funding  agency  —  grantors 

•  personnel  requirements  (e.g.  skill  set  of  personnel) 

•  is  the  program  classified  or  unclassified 

Information  about  Research  Institutions: 

•  research  institution  is  involved  in  (types  of  research) 

•  personnel  (number  of  grad  students,  professors,  etc.) 

•  rating  (e.g.  how  capable  is  the  institution  in  conducting  research  with  respect 
to  a  specific  field) 

o  amount  of  funding  received  for  research  activities 
o  laboratory  capabilities  —  equipment 

•  total  number  of  publications  per  year 

•  international/domestic  collaboration  activities 

•  how  many  grad  students  are  studying  abroad  (e.g.  USA,  Europe) 

•  how  many  grad  students  graduate  per  year 

•  how  many  exchange  students 

•  average  duration  of  Ph.D.  and  M.S.  programs 

Information  about  Development  Institutions: 

•  characteristics  of  national  labs,  (names,  location,  capabilities  to  carry  on 
research) 

•  equipment  -  or  information  about  expensive  equipment,  purchasing  dates  etc. 

•  amount  of  funding  received  /  available 

•  collaboration  activities  (international  and  domestic) 

•  type  of  personnel  associated  with  development  institute-national  lab.  and  their 
skills 

•  number  of  publications  coming  from  the  development  institute  -  national  lab. 

•  sponsorship  of  new  research  activities  (will  give  us  an  indication  if  the 
development  was  stacked  on  something) 

Infonnation  about  Production  facilities: 

•  possible  production  facilities 

locations 

type  of  facility 

rating  -  modernization  level 

collaborations  among  facilities 

•  privately/government  owned  (subcontracting  relationship) 

•  raw  materials  used/on  site 

•  equipment  on  site 
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•  personnel  (number  of  workers) 

•  exporting  capabilities 

•  rate  of  production/quantities 


5.  Model  Metrics 

Our  Fast  Interactive  Integrated  Modeling  and  Strategy  Design  framework  is  capable 
of  generating  an  extensive  list  of  important  metrics  for  the  intelligence  community.  Some 
of  these  metrics  include  perfonnance  against  historical  data,  usability,  confidence, 
timeliness,  and  rapid  identification  of  infonnation  needs.  An  extensive  list  of  metrics  is  as 
follows: 

Quality — measures  of  Accuracy 

-Comparison  with  “expert”  solution; 

-Percent  agreement  between  system  and  analyst; 

-Amount  of  evidence  used  in  analysis 

-Number  of  target  criteria  items  (e.g.  Target  criteria  items,  as  described  in  the  text, 
include  variables,  hypotheses,  documents  found,  etc.  that  experts  identify  as  relevant  and 
important  data  that  should  be  produced — i.e.,  success  criteria.)  considered  by  system 
-Number  of  target  criteria  items  missed  by  system 

Confidence 

-  Analyst  confidence  in  findings 

Presentation  quality  and  clarity 

-  Accessibility:  The  extent  to  which  infonnation  is  available,  or  easily  and  quickly 
retrievable 

-  Concise  representation:  The  extent  to  which  information  is  compactly  represented 

-  Consistent  representation:  The  extent  to  which  information  is  presented  in  the  same 
format 

-  Ease  of  manipulation:  The  extent  to  which  information  is  easy  to  manipulate  and 
apply  to  different  tasks 

-Interpretability:  The  extent  to  which  information  is  in  appropriate  languages, 
symbols,  and  units, 
and  the  definitions  are  clear 

Coverage  and  reliability: 

-Appropriate  amount  of  information:  The  extent  to  which  the  volume  of  infonnation  is 
appropriate  for  the  task  at  hand 

-Believability:  The  extent  to  which  information  is  regarded  as  true  and  credible 
-Completeness:  The  extent  to  which  infonnation  is  not  missing  and  is  of  sufficient 
breadth  and  depth  for  the  task  at  hand 

-Free-of-error:  The  extent  to  which  information  is  correct  and  reliable 

-  Objectivity:  The  extent  to  which  information  is  unbiased,  unprejudiced,  and 
impartial 


Usability  and  analysis  quality 

-Relevancy:  The  extent  to  which  information  is  applicable  and  helpful  for  the  task  at 
hand 

-Reputation:  The  extent  to  which  information  is  highly  regarded  in  tenns  of  its  source 
or  content 

-Timeliness:  The  extent  to  which  information  is  sufficiently  up-to-date  for  the  task  at 
hand 

-Value-Added:  The  extent  to  which  information  is  beneficial  and  provides  advantages 
from  its  use 

The  formulas  used  to  calculate  some  of  the  above  stated  metrics  are: 

Accuracy  =  Number  of  correctly  answered  parts  /  Total  number  of  parts 

Precision  =  Number  of  relevant  results  identified  by  the  subject  /  Number  of  all  results 
identified  by  the  subject 

Recall  =  Number  of  relevant  results  identified  by  the  subject  /  Number  of  relevant  results 
identified  by  the  expert 


6.  Model  Calibration,  Probes  and  Strategy  Design 

Fast  Interactive  Integrated  Modeling  and  Strategy  Design  (FASTIMS)  combines 
several  leading-edge  technologies  developed  by  Icosystem  Corporation  into  an 
innovative,  integrated  framework  for  intelligence  analysis:  In  addition  to  ABM  and 
Network  Model  described  above,  this  framework  also  includes  automated  interactive 
calibration,  sensitivity  analysis,  surrogate  variable  discovery,  and  design  of  scenarios 
(“what-if’)  and  intervention  strategies.  We’ll  briefly  focus  on  our  novel  approach  to 
calibration.  Interactive  Evolutionary  Calibration,  or  IEC. 

Even  when  quality  data  is  available,  it  is  often  the  case  that  only  an  expert  will  be 
able  to  tell  whether  a  model  makes  sense  or  not,  or  tell  apart  two  models  that  fit  the  data 
equally  well.  IEC  works  by  creating  a  range  of  alternative  models,  asking  the  user  for 
feedback,  and  re-creating  a  set  of  models  for  the  user  to  examine,  and  so  forth,  until  the 
user  is  satisfied  with  the  model.  Throughout  this  process,  IEC  always  attempts  to 
maximize  the  model’s  fit  to  whatever  quantitative  data  is  available.  FASTIMS  works 
with  one  user  or  multiple  users.  With  IEC  in  particular,  it  is  a  natural  extension  of  the 
FASTIMS  platfonn  to  involve  a  range  of  analysts,  domain  experts  and  points  of  view  in 
building  and  calibrating  a  model  in  a  collaborative  manner. 


Thus  far  we  have  explored  three  areas  surrounding  the  interactive  evolution  and 
calibration  of  models: 

The  implications  of  uncertainty  in  data  and  probes  for  our  modeling  Efforts 

Specifically,  it  evident  that  we  will  never  have  crisp  enough  data  to  permit  the 
determination  of  a  single  best  model  for  whatever  entity,  like  the  Iranian  government,  is 


9 


under  consideration.  This  observation  highlights  the  critical  role  played  by  our  interactive 
evolutionary/calibration  (IEC)  component.  IEC  will  enable  an  analyst  to  explore  a  space 
of  possible  models  which  are  all  consistent  with  the  current  data  such  that  they  can  use 
their  own  judgment  and  intuition  to  guide  the  selection  of  a  small  number  of  relevant 
models. 

The  use  of  probe  and  strategy  modeling  in  IEC 

Besides  judging  models  on  the  basis  of  their  dynamics,  we  allow  analysts  to  apply  probes 
and  strategies  to  a  space  of  models.  It  is  possible  that  analysts  have  intuitions  about  the 
types  of  strategies  and  probes  to  which  an  entity  like  Iran  will  respond;  they  probably 
also  have  expectations  about  how  that  response  should  look.  By  permitting  analysts  to 
apply  probes  and  strategies  during  the  IEC  step,  we  can  leverage  these  intuitions  and 
expectations  towards  producing  better  models. 

The  pitfalls  of  optimizing  or  calibrating  for  robustness 

If  the  real-world  is  about  to  reach  a  tipping  point,  but  we  optimize  or  calibrate  models  to 
be  robust  with  respect  to  parameter  changes,  then  our  model  might  “smooth  out"  a  real- 
world  transition.  Thus  we  should  be  cautious  about  using  robustness  as  a  criterion  for 
selecting  models.  Steven  Bankes  of  Evolving  Logic,  who  is  working  with  the  Lockheed 
Martin  group,  has  proposed  a  technique  which  potentially  mitigates  this  smoothing  out 
effect.  Bankes'  “Exploratory  Modeling"  integrates  the  behavior  of  many  different  models 
into  a  broader  picture  of  a  system.  If  we  include  models  which  are  not  necessarily  robust 
into  an  exploratory  modeling  framework,  particularly  during  IEC,  then  perhaps  we  can 
mitigate  this  potential  problem  of  “calibrating  away"  a  real-world  tipping  point. 


7  Simulation  Results  and  Future  Research 

The  simulation  scenario  results  demonstrating  capabilities  of  our  integrated 
leadership  and  pathways  model  and  strategy  design  techniques  are  described  in  the 
attached  .pdf  file,  Appendix  1.  The  Agent-Based  model  is  capable  of  testing  different 
hypothesis  regarding  the  nanotechnology  product  intended  use  as  well  as  validating 
variety  of  strategies  designed  in  order  to  influence  the  course  of  development  of 
nanotechnology  based  products. 

We  plan  to  use  these  capabilities  of  the  ABM  of  Leadership  and  Pathways  to  generate 
searchable  space  of  strategies  for  proactive  intelligence.  In  order  to  discover  the  most 
effective  strategies  within  the  searchable  exploration  space,  we  plan  to  apply  optimization 
techniques  based  on  evolutionary  computing  in  our  future  research.  Details  of  this 
approach  will  be  outlined  in  an  upcoming  White  Paper  which  we  are  preparing  for  Dr. 
Peter  Brooks  per  his  request. 
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